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e Aequitas Bias and Fairness Audit Toolkit * Datasheets for Datasets 
Pedro Saleiro, Abby Stevens, Ari Anisfeld, and Rayid Ghani, University Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, 
of Chicago Center for Data Science and Public Policy (2018) Hanna Wallach, Hal Daumé Ill, and Kate Crawford,arXiv:1803.09010v7,(March19, 
http://www.datasciencepublicpolicy.org/projects/aequitas/. 2020),https://arxiv.org/ abs/1803.09010. 
* Diverse Voices Project | A How-To Guide for Facilitating Inclusiveness in * Al Factsheets 360 
Tech Policy IBM Research, https://aif360.mybluemix.net/. 





LassanaMagassa, Meg Young,andBatyaFriedman, Universityof 
Washington Tech Policy Lab, https://techpolicylab.uw.edu/project/ 
diverse-voices/. 
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e Model Cards for Model Reporting 
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy 
Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and 
Timnit Gebru, Proceedings of the 2019 Conference on Fairness, 
Accountability, and Transparency, (2019. 1): 220-229, https:// 
arxiv.org/abs/1810.03993. 





e Al Factsheets 360 
Aleksandra Mojsilovic, IBM Research(2018. 8. 22), 
https://www.ibm.com/blogs/research/2018/08/factsheets-ai/. 


e Al Explainability 360 
IBM Research, https://aix360.mybluemix.net/. 








e Al Fairness 360 
IBM Research, https://aif360.mybluemix.net/. 





e Responsible Machine Learning with Error Analysis 
Besmira Nushi, Microsoft Research 2021. 2. 18), 
https://techcommunity.microsoft.com/t5/azure-ai/responsible- 
machine-learning-with-error-analysis/ba-p/2141774. 








* Aequitas Open Source Bias Audit Toolkit 
Pedro Saleiro, Abby Stevens, Ari Anisfeld, and Rayid Ghani, University of Chicago 
Center for Data Science and Public Policy, 
http://www.datasciencepublicpolicy.org/projects/aequitas/. 
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* FairTest: Discovering Unwarranted Associations in Data-Driven 
Applications 
Florian Tramer, Vaggelis Atlidakis, Roxana G Daniel Hsu, 
Jean-Pierre Hubaux, Mathias Humbert, Ari Juels 2! Huang Lin, 
ArXiv,(2015), https://github.com/columbia/fairtest. 





* Bayesian Improved Surname Geocoding 
Consumer Finance Protection Bureau (2014), https://files. 
consumerfinance.gov/f/201409 cfpb report proxy-methodology.pdf. 
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* Detect Data Drift on Datasets 


BNH.AI, https://www.bnh.ai/public-resources. 





e Watson OpenScale 
IBM, https://www.ibm.com/cloud/watson-openscale. 
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Microsoft Azure Machine Learning 2020. 06. 25), https://docs. 
microsoft.com/en-us/azure/machine-learning/how-to-monitor- 
datasets?tabs=python#create-dataset-monitors. 
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AFrameworkfor Understanding Unintended 
Consequences of Machine Learning 

Harini Suresh and John V. Guttag, arXiv (February 
2020), https://arxiv.org/abs/1901.10002. 





Al Fairness 

Trisha Mahoney, Kush R. Varshney, and Michael Hind, 
O'Reilly (April 2020), https://www.oreilly. 
com/library/view/ai-fairness/9781492077664/. 


Beyond Explainability: A Practical Guideto Managing 
Risk in Machine Learning Models Andrew Burt, Brenda 
Leong, Stuart Shirrell, and Xiangnong (George) Wang, 
Future of Privacy Forum (June 2018), 
https://fpf.org/wp-content, 
uploads/2018/06/Beyond-Explainability.pdf. 





Co-Designing Checklists to Understand Organizational 
Challenges and Opportunities around Fairness in Al 
Michael A. Madaio, Luke Stark, Jennifer 
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CHI “20: Proceedings of the 2020 CHI Conference on 
Human Factors in Computing Systems (April 2020): 1— 
14, https://doi. 
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Closing the Al Accountability Gap: Defining an End-to- 
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Raji, l. D., Smart, A., White, R. N., Mitchell, M., Gebru, 
T., Hutchinson, B., Smith-Loud, J., Theron, D., & 
Barnes, P., FAT”? '20: Proceedings of the 2020 
Conference on Fairness, Accountability, and 
Transparency, (January 2020): 33-44, https://doi. 
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Supervisory Guidance on Model Risk Management 

US Federal Reserve Board (April 2011), https:// 
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in the PublicSector 

David Leslie, The Alan Turing Institute (2019), 
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https://medium.com/design-ibm/the-origins- of-bias-and-how- 
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